Natural-Gradient Stochastic Variational Inference for Non-Conjugate Structured Variational Autoencoder

We propose a new variational inference method which uses recognition models for amortized inference in graphical models that contain deep generative models. Unlike many existing approaches, our method can handle non-conjugacy in both the latent graphical model and the deep generative model, and enables fully amortized inference at test time. Our method is based on an extension of a recently proposed mirror-descent algorithm and employs natural-gradient updates for all three components of the model, i.e. the latent graphical model, the deep generative model, and the recognition model. We also propose structured recognition models to capture posterior correlations among local latent variables. We show that our method has computational advantages over existing approaches in two classes of non-conjugate models, namely, latent mixture models and nonlinear state-space models. An additional advantage of our method is that it can be implemented by reusing existing software for graphical models and deep models.